Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HUDI-8782] BulkInsertWriterHelper parallel close #12518

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

fhan688
Copy link
Contributor

@fhan688 fhan688 commented Dec 18, 2024

Change Logs

close is slow when large amount rowCreateHandles in BulkInsertWriterHelper, this PR close this handles in parallel.

Impact

hudi-flink-datasource

Risk level (write none, low medium or high below)

Low

Documentation Update

None

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@github-actions github-actions bot added the size:S PR with lines of changes in (10, 100] label Dec 18, 2024
}).join();
try {
executorService.shutdown();
executorService.awaitTermination(24, TimeUnit.DAYS);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

24 days?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry! 24 hours. this indicates a long timeout , aiming to waiting for all handles closed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still too long, the close should be fast right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still too long, the close should be fast right?

I agree, close should be fast. here awaitTermination is actually a timeout, 24 Hours is too long , how about wait for a Flink CP interval?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you estimate the normal time cost for the close? Maybe just 1 minute should be enough.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 minute maybe not enough when HDFS is busy. or we can make this a config and set default value 1 minute?

@hudi-bot
Copy link

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:S PR with lines of changes in (10, 100]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants